Weighted Nearest Neighbor Classifiers and First-order Error

Weighted nearest-neighbor classification is analyzed in terms of squared error of class probability estimates. Two classes of algorithms for calculating weights are studied with respect to their ability to minimize the first-order term of the squared error: local linear regression and a new class termed regularized linear interpolation. A number of variants of each class are considered or proposed, and compared analytically and by simulations and experiments on benchmark datasets. The experiments establish that weighting methods which aim to minimize first-order error can perform significantly better than standard k-NN, particularly in high-dimensions. Regularization functions, the fitted surfaces, cross-validated neighborhood size, and the effect of high-dimensionality are also analyzed.